PROCEEDINGS 

OF 

THE ROYAL 
SOCIETY 

rspb.royalsocietypublishing.org 




Review 



8 



CrossMark 

click for updates 



Cite this article: Knight-Jones TJD, Edmond 
K, Gubbins S, Paton DJ. 2014 Veterinary and 
human vaccine evaluation methods. 
Proc R. Soc B 281: 20132839. 
http://dx.d0i.0rg/l 0.1 098/rspb.201 3.2839 



Received: 30 October 2013 
Accepted: 24 March 2014 



Subject Areas: 

health and disease and epidemiology 



Veterinary and human vaccine evaluation 
methods 

T. J. D. Knight-Jones^'2, K. Edmond^ S. Gubbins^ and D. J. Paton^ 

^The Pirbright Institute, Pirbright, UK 

^The Royal Veterinary College (VEEPH), University of London, London, UK 

^School of Paediatrics and Child Health (SPACH), The University of Western Australia, Crawley, Australia 

Despite the universal importance of vaccines, approaches to human and 
veterinary vaccine evaluation differ markedly. For human vaccines, vaccine 
efficacy is the proportion of vaccinated individuals protected by the vaccine 
against a defined outcome under ideal conditions, whereas for veterinary 
vaccines the term is used for a range of measures of vaccine protection. 
The evaluation of vaccine effectiveness, vaccine protection assessed under rou- 
tine programme conditions, is largely limited to human vaccines. Challenge 
studies under controlled conditions and sero-conversion studies are widely 
used when evaluating veterinary vaccines, whereas human vaccines are gen- 
erally evaluated in terms of protection against natural challenge assessed in 
trials or post-marketing observational studies. Although challenge studies 
provide a standardized platform on which to compare different vaccines, 
they do not capture the variation that occurs under field conditions. Field 
studies of vaccine effectiveness are needed to assess the performance of 
a vaccination programme. However, if vaccination is performed without 
central co-ordination, as is often the case for veterinary vaccines, evaluation 
will be limited. This paper reviews approaches to veterinary vaccine 
evaluation in comparison to evaluation methods used for human vaccines. 
Foot-and-mouth disease has been used to illustrate the veterinary approach. 
Recommendations are made for standardization of terminology and for 
rigorous evaluation of veterinary vaccines. 
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1. Introduction 

Vaccines are crucial in the control of many human and veterinary diseases. 
Routine vaccination is used by most countries in the world to control about 
15-20 human infectious diseases, and roughly another 15 diseases are selectively 
targeted [1]. It is estimated that veterinary vaccines are available for over 400 
diseases affecting mammals, birds and fish, including farm animals, pets and 
wildlife [2]. Though revenues from the global human vaccine market are over 
30 times that of veterinary vaccines [2], veterinary vaccines are very widely 
used with over two billion doses of foot-and-mouth disease (FMD) vaccine 
used per year [3], and poultry vaccines given on an even greater scale [4]. 

Given the enormous scale and implications of vaccine use in terms of both 
health and economics, it is clearly important that their effectiveness be thoroughly 
evaluated. In fact, human and veterinary vaccines are evaluated in very different 
ways. Here, we review the various approaches to vaccine evaluation and discuss 
both the rationale for these different approaches and the problems encountered. 
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(a) Fundamental differences 

Despite the large number of veterinary vaccines in use, the literature on 
their evaluation is small compared with that in the human vaccine field; this 
is exacerbated by a failure to publish findings by vaccine manufacturers. 
A basic terminology exists employing the words efficacy, effectiveness and 
coverage; yet, these terms are inconsistently used in the veterinary world 

© 2014 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution 
License http://creativecommons.0rg/licenses/by/3.O/, which permits unrestricted use, provided the original 
author and source are credited. 



Table 1. Definition and usage of terms in human and veterinary vaccine evaluation. 



term 




definition 1 


vaccine 
potency 


veterinary 


'relative strength of a biological product as determined by appropriate test methods. (Initially the potency is 
measured using an efficacy test in animals. . . with pathogen challenge;. . . later this may be correlated with 
tests of antigen content, or antibody response, for routine batch potency tests.)' [5] 




human 


'potency is the specific ability or capacity of the vaccine as measured by a laboratory test' [6] 




difference 


similar definition but less frequently used for human vaccine evaluation 


vaccine efficacy 


veterinary 


'specific ability of the biological product to produce the result for which it is offered when used under the 
conditions recommended by the manufacturer' [5] 






'the ability of the vaccine to give protection against the adverse effects of the infection to the vaccinated 
animal...' [7] 




human 


'...the percentage reduction in disease incidence attributable to vaccination [usually] calculated by means of the 
following equation: 

VL [70) = [K[i — K\j)/K[i X lUU, il. Ij 

where /?u = the incidence risk or rate in unvaccinated people and Ry = the incidence in vaccinated people . . . 

measured in an individually randomized placebo-controlled clinical trial' [1]. 
The equation for vaccine efficacy can be reformulated as: 

Vb = 1 — Ky/K[}, (l.zj 

where /?v/Ru the relative risk or rate ratio. 




difference 


veterinary usage has not been standardized 


vaccine 


veterinary 


usually not a specific term, more the ability of a vaccine to control disease in the field [8] 


effectiveness 


human 


vaccine efficacy measured by observational studies under field conditions within a vaccination programme [1] or 
measured by trials conducted under normal programme conditions 


correlate of 


veterinary 


a variety of terms are used to describe this widely used concept 


protection 


human 


a specific response to a vaccine that is associated with protection against infection, disease, or other defined 
endpoint [9,10] 


vaccine 
coverage 


veterinary 


as for human — although occasionally it refers to the proportion of the target population that have sero-converted 
to a protective titre; the latter is sometimes called immunization coverage or population immunity [11] 




human 


the proportion of the target population that have been vaccinated according to a defined schedule. Sometimes 
called immunization coverage [12] 




difference 


occasionally in veterinary programmes Immunization coverage' may refer to the proportion that have sero-converted 



above a titre deemed protective 



(table 1). This is partially explained by the range of disease 
outcomes targeted by livestock vaccines to improve 
profitability (see examples in table 2). 

Although herd effects may be considered [13], the out- 
come of interest for human vaccine evaluations is typically 
the status of the individual. In veterinary medicine, assess- 
ment of overall group status is common, as management is 
often done at the group level. This results in the lack of indi- 
vidual data and analysis. This extends to disease control, 
where spread between herds may be of greater concern 
than spread within already infected herds. However, there 
are potential problems with evaluation at the group level, 
including a failure to account for population turnover 
and variation in immunity and pathogen exposure within 
a group. These inaccuracies and confounders can lead to a 
limited or incorrect understanding of vaccine protection. 

For certain notifiable animal diseases, zonal or national 
disease-free status is required to gain access to lucrative 



international export markets for animals and their products. 
This leads to a focus on regional pathogen eradication. For 
human disease control, the emphasis is on reducing morbid- 
ity regardless of infection status. Exceptions include 
elimination programmes (e.g. polio) and novel malaria vac- 
cines that block transmission. Options for restricting contact 
between infected and susceptible humans are limited, 
depending instead on immunity to control disease. This dif- 
fers from the control of important veterinary diseases 
where the use of culling and movement controls is long- 
established. Furthermore, immunization may be prohibited 
where disease-free status must be proved and it is not 
possible to distinguish vaccinated and infected animals. 

As livestock are ultimately economic commodities, besides 
welfare considerations, disease control must be profitable. 
This limits resources available for veterinary vaccine develop- 
ment and application. Human vaccines may cost more than 
US$100 per dose, by contrast an individual chicken is worth 



Table 2. Examples of disease outcomes targeted by veterinary vaccines, other than clinical disease in vaccinated animals. 



outcome 


disease examples 


mortality 


clostridial diseases, rinderpest, cattle lungworm 


abortion rate 


porcine parvovirus and porcine reproductive and respiratory syndrome (PRRS). Infectious bovine 
rhinotracheitis, bovine viral diarrhoea (BVD) and salmonellosis in cattle. Chlamydophila abortus and 
toxoplasmosis in sheep. Equine herpesvirus 


weight gain and efficiency of feed 
conversion into meat 


PRRS, porcine circovirus 


Qisease transmission rates 


various inciuaing porcine circovirus ana riviu 


shedding of zoonotic pathogens (to 
protect human health) 


Salmonella enteritidis in hen eggs, Escherichia coli 0157 in cattle 


morbidity in offspring after vaccination of 
dams 


various including rotavirus and E. coli in cattle 


protection against fetal infection in utero 


BVD 



b 
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only a few dollars. Although higher prices may be paid for pets 
and breeding stock, the size of the market is small. 



(b) Stages of evaluation 

A vaccine may be evaluated during development, licensing 
and introduction, vaccine batch testing, programme monitor- 
ing or after suspected vaccine failure. To obtain licensure, 
human vaccines typically undergo initial evaluation in 
animal models followed by a series of controlled trials (phase 
I, II and III; electronic supplementary material, table SI) with 
an increasing number of human subjects to assess safety, 
immunogenicity and then efficacy (defined in table 1) [1]. 
National health ministries then evaluate such information 
before allowing vaccines to be introduced. If a vaccine is to 
be used in a state-funded programme, cost-effectiveness will 
also be evaluated. After licensure and introduction of a 
human vaccine, protection in the field against natural challenge 
is estimated by means of observational studies and called 
vaccine effectiveness (phase IV) [12]. 

In order for veterinary vaccines to obtain market authoriz- 
ation, they are subjected to safety and immunogenicity studies 
on a limited number of individuals of the target species 
[14-17]. Their ability to protect is assessed by in vivo challenge 
or occasionally by sero-conversion studies, the results of 
which have been expressed using a variety of different statis- 
tics often called measures of efficacy [5,14]. Although they 
are used in the assessment of efficacy, the scale of veteri- 
nary vaccine field studies are limited compared with human 
vaccine trials. 

Only on rare occasions are field studies not required at 
all for licensure of a veterinary vaccine in the European 
Union, e.g. when pathogen challenge in the field is unreliable 
or when rapid licensure is required during an emergency 
situation. Field trials may not be possible when vaccination 
is prohibited, as is often the case for exotic notifiable 
animal diseases in countries needing to prove free status 
[14,16,17]. However, in less regulated parts of the world, 
field studies play a very limited role in veterinary vaccine 
authorization and are typically used to evaluate safety 
rather than efficacy [5]. 



2. Evaluating protective effects in vaccinated 
humans and animals 

Below, we consider a hierarchy of studies employed for the 
evaluation of human and veterinary vaccines (summarized 
in table 3). These different designs vary in both the value of 
evidence that they provide and their resource requirements. 

(a) Challenge studies 

Vaccinated and unvaccinated individuals may be compared 
after direct challenge with the target pathogen under controlled 
experimental conditions. Challenging humans with dangerous 
pathogens is rarely acceptable. However, challenge studies 
using animal models are important for the initial evaluation of 
human vaccines. Challenge studies with human subjects are 
sometimes performed for pathogens with effective treatments, 
such as specific strains of malaria, and where the disease is 
usually self-limiting, such as typhoid, cholera, the common 
cold and influenza [18]. Human challenge is also used to 
assess protection against an attenuated pathogen, for example 
oral polio vaccine [19]. 

The evaluation of veterinary vaccines relies heavily on 
challenge studies. Typically, protection is assessed using a 
high level of pathogen challenge with the lowest vaccine anti- 
gen content permitted under the authorization. Although this 
will provide some confidence that the vaccine will protect 
even in extreme situations, the controlled conditions of a chal- 
lenge study will not reflect the sometimes suboptimal 
application of vaccines in the field. For some important veter- 
inary pathogens, the design of these challenge studies is 
prescribed by official standards (box 1) [5,16]. 

Owing to concerns about animal welfare, cost and laboratory 
pathogen escape, the number of animals used for challenge 
evaluation is generally small and the length of foUow-up limited. 
Consequently, results can be statistically uncertain. 

Challenge studies allow a high level of control over charac- 
teristics of the participants and pathogen exposure, minimizing 
differences between vaccinated and control groups. Accurate 
and detailed outcome measures can improve the statistical 
power when sample size is small. 



Table 3. The differing evaluation methods for human and veterinary vaccines. 



evaluation method 


human usage 


veterinary usage 


challenge studies 


initial evaluation with animal models 
subsequent human challenge studies are performed for 
certain pathogens 


initial and final vaccine efficacy testing 


randomized trials 


individual and cluster randomized trials routinely used 


usually used for licensure efficacy 




for licensure efficacy evaluation 


evaluation 




sometimes for post-licensure effectiveness evaluation 




post-vaccination immune correlate response 


often used pre-licensure and occasionally for licensure 


often used pre and post-licensure 


vaccine effectiveness observational field studies 


routinely used for monitoring post-licensure 


rarely performed 


vaccine effectiveness observational studies 


routinely used for monitoring post-licensure when 


not performed 


using routine surveillance data 


adequate data are available 




post-vaccination sero-conversion field surveys 


rarely used 


often used for monitoring post-licensure 


sero-prevalence population immunity surveys 


rarely used 


often used for monitoring post-licensure 


in vitro serological matching assays 


used post-licensure when suitable assay exists 


often used post-licensure when suitable 
assay exists 


coverage evaluation 


various methods routinely used 


distributed method sometimes used 



Box 1. FMD 50% protective dose (PD50). 

In Europe, FMD vaccines are routinely evaluated using the PD50 test. 

Three groups of at least five cattle are given different doses of vaccine (typically a full, a quarter and a 16th dose). Two 
unvaccinated control animals are also used. After three to four weeks, animals are given a standard dose of FMD virus 
injected into the tongue. Animals are observed for foot lesions. From these data, the fraction of the standard dose of vaccine 
that would protect 50% of exposed cattle is then estimated. The reciprocal of this is the PD50 value [5,16]. This is a measure of 
vaccine potency, reflecting protective efficacy. 



Challenge studies provide a standardized platform on 
which to compare different vaccines for the same disease or 
to compare the effect of specific variables on vaccine protection. 
However, the challenge may not mimic natural pathogen 
exposure and under field conditions many factors will vary in 
ways which are not captured. 

(b) Randomized controlled trials 

This approach is used more routinely in the evaluation of 
human vaccines. In a randomized controlled trial (RCT), 
a study group that represents the population of interest is 
identified, preferably with a high incidence of the disease. Indi- 
viduals within this population are then selected at random to 
be vaccinated, or to receive either no vaccine, a placebo or an 
alternative vaccine. This latter point is important as people 
may act differently if they think they have been vaccinated. 
Vaccine storage and delivery is done exactly according to the 
manufacturer's instructions. The protective efficacy of the vac- 
cine can then be calculated by comparing the incidence in the 
vaccinated and control groups (equation (1.1), table 1). 

RCTs are often referred to as the 'gold standard' for asses- 
sing the effect of public health interventions [20]; one reason 
being that vaccinated and control groups have similar levels 
of exposure to all known and unknown confounding risk 
factors owing to the randomization process. 



As well as being used for pre-licensure phase III trials (see 
the electronic supplementary material, table SI), national 
health agencies may perform RCTs to evaluate the likely effi- 
cacy of a particular vaccine schedule. Sometimes trials are 
performed under programmatic conditions to obtain estimates 
of field protection, called vaccine effectiveness rather than 
efficacy, the latter being measured under ideal conditions. 

The European Medicines Agency (EMA) specifies guidelines 
and standards for RCT designs for veterinary vaccines [21]. 
Although field trials are used for veterinary vaccines, unlike 
human medicine, they are sometimes thought of as inferior 
methods of efficacy evaluation compared to the standardized 
and highly controlled conditions of the challenge study [5]. In 
addition, the cost associated with large trials poses a problem 
for some veterinary vaccines for which the market is relatively 
small [2,4]. Furthermore, as entire groups of livestock are typi- 
cally vaccinated at the same time, cluster randomized designs 
(considered later) may be more relevant than trials, where 
vaccinated and unvaccinated individuals exist in the same herd. 

(c) Vaccine effectiveness evaluation: observational 
studies 

Observational studies are the main method of evaluating human 
vaccines once used in the population at large [12,22-24]. This 



approach has been neglected in animal populations, although 
there are some examples of its use [25,26]. 

Several different observational study designs exist (some 
key designs are described later). Most calculate the vaccine effec- 
tiveness statistic based on the standard formula (equation (1.1), 
table 1) [6]. For these studies, the term vaccine effectiveness is 
used, denoting that the evaluation is of vaccine performance 
under programmatic conditions where vaccine storage, delivery 
and participant health status will vary. 

In an RCT, a vaccine is administered to individuals chosen 
at random. This is not the case for observational studies, 
where vaccinated individuals are likely to differ from those 
not vaccinated in ways that may confound the vaccine effect. 

(i) Cohort studies 

In a cohort study, incidence (risk or rate) is compared in vacci- 
nated and unvaccinated groups over the period of observation. 
Controlling for differing levels of pathogen exposure is vital 
and sometimes challenging. In some cohort studies, only indi- 
viduals from affected subgroups or households are included 
(household secondary attack rate study); the assumption is 
made that individuals living in the same house as a case receive 
a similar pathogen exposure [24]. 

Compared to a prospective study, conducting a study retro- 
spectively increases the chance of obtaining incorrect data as 
the passing of time and outcome status may affect recall. 

Cohort studies are often used to evaluate human vaccines, 
sometimes as part of large, ongoing studies [27] or during 
opportunistic, retrospective analysis of an outbreak [28]. 
Where national databases with health records for all individ- 
uals exist they can be used for national studies of vaccine 
effectiveness [29]. Large cohort studies are less common for 
livestock, partly because of cost. Retrospective studies, using 
either farm records or after outbreaks among small-holders, 
are more feasible [26]. 

(ii) Case -control studies 

It is also possible to estimate vaccine effectiveness by compar- 
ing prior vaccination status of affected individuals with the 
vaccination status of controls that were similarly exposed, 
but failed to contract the disease [30]. Vaccine history is col- 
lected retrospectively and confounders must be adjusted for. 

This is a common method of human vaccine effectiveness 
evaluation. As it is relatively quick and inexpensive to perform 
[31], the method would be suitable for veterinary vaccines pro- 
vided that accurate vaccination and disease data are available. 
However, the lack of vaccinated and unvaccinated animals on 
the same premises and increased likelihood of vaccination in 
high-risk groups may prevent identification of a suitable control 
group. The method may also not be possible within a highly 
effective control programme owing to the lack of cases. 

(iii) Vaccine programme impact 

A change in vaccination strategy may be assessed in a vaccine 
impact study by comparing disease burden within a popu- 
lation or cohort before and after the change [32]. Potential 
bias from underlying temporal trends must be considered; 
also such studies require good pre- and post-vaccination dis- 
ease surveillance to accurately detect changes in disease 
burden. A fall in incidence could be because of vaccine effect 
or some other factor. If incidence does not fall, the programme 
is not achieving its objectives either due to low vaccine 



coverage or effectiveness, although increases in other drivers 
of disease may have coincided with vaccination. 

This problem can be overcome to some extent if vaccine 
implementation is phased in over time rather than all at 
once, allowing contemporaneous comparison of vaccinated 
and control populations. Confounding is further controlled 
in a 'stepped wedge design', where the vaccine is introdu- 
ced in several steps. By randomly selecting which regions 
are included in each step, vaccinated and yet to be vaccinated 
regions are balanced in terms of confounders (see Cluster 
randomized trials section) [33]. 

Changes in incidence are routinely assessed in both 
human and veterinary vaccination programmes, sometimes 
correlating incidence with coverage. Sero-prevalence surveys 
are often used for livestock as an unbiased measure of disease 
burden where under-reporting is a problem. However, sero- 
positivity owing to infection must be distinguishable from 
vaccine-induced sero-positivity. 

The burden of disease prevented by a human vaccine 
is sometimes estimated as a function of vaccine coverage, 
vaccine effectiveness and pre-vaccination disease incidence. 

(iv) Relative effectiveness 

The level of protection afforded by a vaccine can be compared 
to that of another vaccine or a different schedule to give an 
estimate of relative effectiveness. Many of the above studies 
(including RCT) can be adapted for this situation. 

(v) Outbreak studies 

Many of these study designs are based on observations made 
during outbreaks, often through retrospective analysis. When 
there is a lack of unvaccinated animals, inadequate protection 
may be identified by outbreaks in vaccinated populations 
without comparison to a control group. However, it may be 
difficult to quantify the level of vaccine effectiveness. This 
may be the case when evaluating outbreaks in commercial 
farms with uniform management. 

Evaluation of reactive vaccination performed in response 
to outbreaks can be challenging as the investigator may be 
unsure if individuals were already immune before vacci- 
nation, challenge may occur before vaccinated individuals 
have responded to the vaccine and those left unvaccinated 
may have a different risk of pathogen exposure. 

(d) Serological evaluation 
(i) Correlates of protection 

Vaccines often induce a measurable response (e.g. antibody 
titre). If this response is correlated with protection against dis- 
ease or infection it can be used as an alternative outcome for 
vaccine evaluation [1,9,10]. Correlates of protection are 
widely used for both human and veterinary vaccines [34]. 

In recent times, certain human vaccines, notably menin- 
gococcus C in the UK [35] and meningitis A vaccine in 
Africa [36], have been licensed based on serological correlates 
of protection without a stage III RCT with the proviso that 
close monitoring of vaccine effectiveness is performed after 
introduction of the vaccine. There is pressure to minimize 
the use of animal challenge studies [37], evaluating serologi- 
cal measures of protection instead [38] (box 2). Although 
serological studies are routinely used in the evaluation of 



Box 2. Correlates of protection — expected percentage of protection (EPP). 

The EPP is a standardized test used to assess the potency of FMD vaccines using serology rather than pathogen challenge. In 
this method, the sera from 16 to 30 cattle between 18 and 24 months of age, taken 30 days post-vaccination are assessed for 
their ability to neutralize or bind virus (typically the vaccine strain) using a virus neutralization (VN) test or an ELISA. The 
proportion of animals expected to be protected is then estimated by referring to serological titres and observed protection 
from multiple previous challenge studies [5,39]. 



Box 3. r-value test. 

The V-value' is an in vitro assay of FMD vaccine match; this is a measure of the relative reactivity of sera from vaccinated 
cattle to the field virus in question compared to the reactivity of the same sera to the virus strain used to make the vaccine, 
performed by ELISA or VN [5,39]. 

For FMD, a suboptimal vaccine match may be compensated for by having a more potent vaccine that stimulates greater 
antibody production, e.g. one that contains more antigen per dose [39,44]. The test provides rapid results, but there can be 
problems with test repeatability [39] and results do not tell you whether the vaccine is actually protecting animals in the field. 



veterinary vaccines, sero-conversion per se is rarely used as a 
measure of efficacy during licensure. 

Many traditional livestock systems do not keep written or 
computer records. Although some modern commercial farms 
keep excellent individual animal production records, many 
do not. This is due to the large number of individuals kept 
on one farm, the limited value of individual animals and the 
high rates of population turnover and movements. As sero- 
status can act as a record of prior infection or present immunity, 
it is widely used in veterinary settings. However, even if vacci- 
nation status is known, with a single serum sample it may 
be impossible to tell whether infection came before or after 
vaccination limiting its use for efficacy estimation. 

(ii) Post-vaccination sero-conversion surveys 

In human vaccination campaigns, sero-conversion studies are 
sometimes performed using pre- and post-vaccination sera to 
assess vaccine response [23]. This is typically used for phase 
II immunogenicity trials. 

Similar surveys using only sera collected post-vaccination 
are common in livestock. The proportion with an antibody 
titre above a specified threshold associated with protection 
is then determined [40]. 

(iii) Sero-prevalence surveys 

This involves assessing sero-status for a representative sample 
of the population irrespective of vaccination status after a 
vaccination campaign [23]. Not widely used for human vac- 
cines, these surveys are used in veterinary settings to assess 
the level of 'population immunity' [41], under the assumption 
that sero-positivity implies protection. Sero-prevalence is a 
function of the proportion vaccinated, the proportion that 
sero-convert post-vaccination and the proportion sero-positive 
following natural infection. In endemic populations, it is there- 
fore difficult to infer if high levels of sero-positivity reflect high 
coverage with an effective vaccine or widespread infection, or a 
combination of both vaccination and infection. Where vaccine 
protection is short-lived or population turnover is rapid, 
studies need to be regularly updated. 

Vaccine effectiveness studies require pathogen exposure 
and so can only be implemented once outbreaks have 



occurred, by which time it may be too late to affect the out- 
come. Assessing population immunity via sero-surveys can 
detect susceptibility before outbreaks occur; this is useful 
where vaccination is used in disease-free populations, par- 
ticularly when reliable estimates of vaccine coverage and 
effectiveness are not available. This method has proved 
useful for veterinary vaccines and could aid the evaluation 
of human population immunity when vaccine records are 
poor and a measureable correlate of protection exists. 

(iv) In vitro vaccine matching assays 

The likely performance of vaccines may sometimes be pre- 
dicted via in vitro serological methods. Antigenic match 
between influenza vaccines and field viruses is assessed 
using sera from vaccinated ferrets, or sometimes people, 
measuring the sera's ability to react with the field viruses 
[42]. However, these matching studies do not consistently 
predict effectiveness; currently, the same is true for alter- 
native predictors of match based on genetics [43]. A similar 
veterinary assay is the 'r-value' [39] (box 3). 

(v) Cell-mediated correlates of immunity 

Although most correlates of protection measure the humoral 
immune response, cell-mediated immune response is increas- 
ingly assessed in human vaccine studies. This is primarily 
for intracellular infections such as tuberculosis [45]. Assays 
typically assess the antigen-specific response of T cells 
(e.g. ELISpot) and their associated cytokines, such as IFN-7, 
used to evaluate BCG vaccination in humans [46], cattle [47] 
and badgers [48]. 

Combining information on vaccine potency and antigenic 
match improves the prediction of efficacy [44] with identification 
of genetic predictors under development [49]. 

(e) Direct versus indirect effects of vaccination 

Direct vaccine protection is the reduction in risk in vaccinated 
compared with similarly exposed unvaccinated individuals. 
However, vaccinating some but not all members of a group 
can result not only in protection of those vaccinated, but 
also reduced pathogen exposure and morbidity in those not 
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Figure 1. Diagram showing the different types of vaccine effect detectable in a cluster trial and which vaccine groups to compare to estimate them. Within a cluster, I/and 
U represent vaccinated and unvaccinated individuals, respectively [6]. Using this design, the different effects (direct, indirect, total and overall) can be estimated by 
comparing groups as indicated by the arrows. Coverage in the vaccinated cluster is <100%. 



vaccinated. This indirect vaccine effect is due to a reduction in 
transmission w^ithin the group as a w^hole. Studies that only 
capture the direct effect of vaccination by comparing vacci- 
nated and unvaccinated individuals in the same group 
may underestimate the overall effect of vaccination by not 
capturing the indirect effects. 

(i) Cluster randomized trials 

In cluster randomized trials (CRTs), the intervention is ran- 
domly allocated to entire clusters, rather than individuals. 
Certain CRTs (and observational vaccine effectiveness studies) 
can be designed so as to capture direct and indirect vaccine 
effects (figure 1). 

By randomizing allocation to different clusters, rather 
than individuals w^ithin the same cluster, inferences can be 
made on the overall effect of vaccination on a community, 
rather just the direct effects afforded to the individual 
[13,22]. Vaccinated and control clusters w^ill tend to be similar 
due to randomization. 

CRTs are increasingly seen as the most relevant study 
design for informing policy and are w^idely used to evaluate 
human vaccines. The group management of production ani- 
mals naturally lends itself to this study design and, although 
the term CRT is rarely used, the method is common, especially 
for fish and poultry [50,51]. How^ever, CRTs have seldom if 
ever been conducted for veterinary vaccines used in national 
control programmes of notifiable diseases such as FMD, 
brucellosis or peste des petits ruminants. 



3, Vaccine coverage 

As well as evaluating w^hether vaccinated individuals are 
protected, it is crucial to check that a sufficient proportion 



of individuals receive the vaccine as per the vaccine schedule 
(i.e. vaccine coverage). There are several w^ays of assessing 
coverage (see below^) [12]. Although the most costly often 
provide the best data, they may not alw^ays be necessary. 

(i) Distributed method: the simplest approach is to deter- 
mine the number of doses distributed divided by the 
target population size. But this does not reveal if 
individuals received the full course or account for 
w^asted/ unused doses. Inaccurate estimates of target 
population size w^ill bias coverage estimates. 

(ii) Administered method: examining the number of doses 
actually administered from central or local records 
can provide more accurate and detailed measures 
of coverage. 

(iii) Surveys: surveys may be based on convenience 
samples (e.g. schools), though the population sampled 
may not be representative of the population at risk. 
Alternatively, structured surveys can be implemented, 
for example the WHO 30 cluster, tw^o-stage stratified 
random survey [52]. 

(iv) Sero-prevalence: finally, coverage may be partially 
inferred from sero-prevalence surveys (see above), 
but difficulties distinguishing prior infection from 
vaccination may exist. 

The recommended method of coverage monitoring will 
vary depending on the setting, for example, in areas w^ith 
only w^ell-organized commercial farms it may be satisfactory 
to use routinely recorded data to monitor coverage at the 
herd level assuming that vaccination is then applied to all 
eligible animals w^ithin a herd. In other settings, this assump- 
tion w^ill be incorrect and routinely recorded data may not 
be reliable. 



4. Discussion 

Under field conditions, the performance of both human and 
veterinary vaccines can vary unexpectedly. There are various 
reasons for this, including vaccine factors, such as variable 
batch potency, poor administration, failure to observe shelf- 
life and cold chain requirements; pathogen factors such as 
level of challenge and the emergence of novel field strains 
with poor vaccine match; and host and environmental factors 
that influence immune response, such as genetics and nutrition. 
Furthermore, population density and nature and frequency of 
contacts will influence level of challenge. Often variation in pro- 
tection cannot easily be explained, let alone predicted (e.g. 
[30,53]). Without ongoing vaccine evaluation, including moni- 
toring of effectiveness and coverage, it will be difficult to 
anticipate and explain breakdowns in disease control within a 
vaccination programme. 

In the field, a vaccine will have to protect individuals 
of differing susceptibility and pathogen exposure level. 
Human and veterinary medicine deal with this in very differ- 
ent ways. Field trials evaluating human vaccines are designed 
to include much of this variation. If a vaccine is to be used in 
a setting different to previous trials, further studies may be 
conducted. The protective effect of a veterinary vaccine is 
often assessed in small studies which minimize variation in 
animal susceptibility and exposure level. In theory, veterinary 
vaccines are then formulated with a potency that is expected 
to protect even when animal susceptibility and pathogen 
exposure are high [34]. Despite this, vaccine failure in the 
field can still occur. 

The authorization process is fundamentally different for 
human and veterinary vaccines. Human vaccine licensing 
is based on limited controlled laboratory studies and exten- 
sive clinical trials and field effectiveness studies. Veterinary 
vaccines are authorized on the basis of more extensive con- 
trolled laboratory studies involving pathogen challenge, 
backed up (usually) by field studies which are less extensive. 
Human health is largely overseen by public bodies with 
funds available for large field trials, whereas animal health 
is largely dealt with by the private sector which has a limited 
capacity to fund and coordinate extensive vaccine evaluation 
studies. Unlike for key human diseases, incidence, prevalence 
and antigenic change are rarely monitored systematically to 
inform veterinary vaccination policy. 

As governmental interest in livestock health is largely lim- 
ited to notifiable diseases, most countries lack coordinated 
control programmes for endemic, non-notifiable diseases 
that cause ongoing losses to the livestock sector. By contrast, 
ministries of health try to limit the impact of all human dis- 
eases, both endemic and exotic. As disease control on one 
farm affects the disease risk faced by others, central coordi- 
nation is required for a programme to be effective. In 
developed countries, governmental veterinary vaccination pro- 
grammes typically concern the short-term control of outbreaks 
of exotic or emerging pathogens (e.g. FMD, bluetongue). For 
endemic veterinary diseases, vaccination may be applied routi- 
nely, often with inconsistent evaluation and limited ability to 
adapt to the situation on the ground; this is particularly true 
where vaccination is left to the private sector. By contrast, the 
enormous reduction in vaccine preventable childhood diseases, 
seen over the past 50 years, would not have occurred without 
central coordination despite the enormous interest parents 
have in the health of their children. 



(a) Possibilities for veterinary vaccine effectiveness 
evaluation 

Despite its importance, the area of effectiveness evaluation has 
been little explored for veterinary vaccines. The under-utilization 
of vaccine effectiveness studies in the veterinary sector becomes 
even more apparent when one considers the ever-increasing 
pressure to reduce the use of animals in experimental studies. 
So far in the veterinary field, this has largely been addressed 
through the use of correlates of protection without supporting 
evidence from studies of protection in the field. 

Organizations involved in drug authorization, including 
the EMA, are currently considering whether pharmaceuticals, 
including vaccines, could be licensed using data for a limited 
range of indications with the possibility of adding further 
indications later on using post-authorization studies. How- 
ever, in the veterinary field, this has been hampered by the 
lack of established methodology for vaccine effectiveness 
evaluation. To assist researchers interested in this area, rec- 
ommended methods for doing this have been described in 
the electronic supplementary material. 



5. Conclusion 

The importance of independent vaccine evaluation including 
quality assurance cannot be emphasized enough. Both vac- 
cine evaluation and quality assurance were crucial for the 
global eradication of smallpox and rinderpest. The cost of 
thorough evaluation can be justified when one looks at the 
huge overall cost of vaccination programmes, the uncertainty 
that often exists about effectiveness and the major benefits 
experienced when programmes are successful. 

In order to evaluate vaccine programmes in the field, a 
number of challenges need to be addressed. These include 
the paucity of individual disease and vaccine records and dif- 
ficulties in finding appropriate vaccinated and unvaccinated 
comparison groups. When suitable records are not available 
but a good correlate of protection is known, sero-surveys 
have provided useful estimates of livestock population 
immunity. This evaluation method may be appropriate for 
human vaccines when effectiveness studies are not possible. 

Vaccine effectiveness studies are essential for measuring pro- 
tection actually achieved within a vaccination programme. 
Cluster trials performed under ordinary field conditions are par- 
ticularly informative [22]. Adoption of these methods by the 
veterinary sector would provide a better understanding of the 
full benefits and costs of vaccination. This evidence base would 
help to secure funding for effective disease control and leave 
less room for speculative policy-making. However, evaluation 
of protection at the population level requires central coordination. 

Acknowledgements. The authors thank Prof. Paul Fine (London School of 
Hygiene and Tropical Medicine) for his help in initiating the com- 
parison between approaches used for studying human and 
veterinary vaccine effectiveness and Dr Keith Sumption (European 
Commission for the Control of FMD) for facilitating studies looking 
at the field evaluation of FMD vaccination programmes. The authors 
are extremely grateful to Prof. David Mackay at the European Medi- 
cines Agency and Dr Noemi Garcia del Blanco at the UK Veterinary 
Medicines Directorate for their informative advice and feedback. 
Funding statement. T.J.D.K-J. is supported by the Biotechnology and Bio- 
logical Sciences Research Council (BBSRC) and by the European 
Commission for the Control of FMD. S.G. and D.J.P. acknowledge 
funding from BBSRC and D.J.P. is a Jenner Investigator. 



References 



10. 



12. 



13. 



14. 



15. 



16. 



Plotkin SA, Orenstein WA, Offit PA. (eds) 2008 
Vaccines, 5th edn. Philadelphia, PA: Saunders. 
Van Aarle P. 2010 Immunological correlates of 
vaccine-derived protection against FMD: the 
regulatory perspective. In Vaccine efficacy: 
immunological correlates of vaccine derived 
protection, Fondation Merieux, Veyrier-du-Lac, 
France, 20-22 September 2010. 
Knight-Jones TJ, Rushton J. 2013 The economic impacts 
of foot and mouth disease — what are they, how big are 
they and where do they occur? Prev. Vet Med 112, 
161 - 173. (doi:10.1016/j.prevetmed.2013.07.013) 
Meeusen ENT, Walker J, Peters A, Pastoret P-P, 
Jungersen G. 2007 Current status of veterinary 
vaccines. Clin. Microbiol. Rev. 20, 489-510. (doi:10. 
1128/cmr.00005-07) 

01 E. 2012 Manual of diagnostic tests and vaccines 

for terrestrial animals 2012. Paris: World 

Organisation for Animal Health. 

Halloran ME, Longini IM, Struchiner CJ. 2009 Design 

and analysis of vaccine studies. New York, NY: 

Springer. 

Pastoret P-P. 1997 Veterinary vaccinology. 

Amsterdam, The Netherlands: Elsevier. 

Parida S. 2009 Vaccination against foot-and-mouth 

disease virus: strategies and effectiveness. Expert 

Rev. Vaccines 8, 347-365. (doi:1 0.1 586/1 4760584. 

8.3.347) 

Plotkin SA. 2008 Vaccines: correlates of vaccine- 
induced immunity. Clin. Infect Dis. 47, 401-409. 
(doi:10.1086/589862) 

Nguipdop-Djomo P, Thomas SL, Fine PEM. 2013 

Correlates of vaccine-induced protection: methods 

and implications. WHO/I VB/1 0.00, 181. See http:// 

apps.who.int/iris/bitstream/10665/84288/1/ 

WH0JVB_13.01_eng.pdf. 

Hu RL, Fooks AR, Zhang SF, Liu Y, Zhang F. 2008 

Inferior rabies vaccine quality and low immunization 

coverage in dogs {Canis familiaris) in China. 

Epidemiol. Infect. 136, 1556-1563. (doi:10.1017/ 

S0950268807000131) 

Chen RT, Orenstein WA. 1996 Epidemiologic methods 
in immunization programs. Epidemiol. Rev. 18, 
99-117. (doi:10.1093/oxfordjoumals.epirev.a017931) 
Millar EV, Watt JP, Bronsdon MA, Dallas J, Reid R, 
Santosham M, O'Brien KL. 2008 Indirect effect of 
7-valent pneumococcal conjugate vaccine on 
pneumococcal colonization among unvaccinated 
household members. Clin. Infect. Dis. 47, 989-996. 
(doi:10.1086/591966) 

European Medicines Agency. 2003 Committee for 

veterinary medicinal products: revised position 

paper on indications for veterinary vaccines-EMEA/ 

CVMP/042/97-Rev.1-FINAL. 

USDA. 2011 Veterinary Services Memorandum NO. 

800.50. Animal and Plant Health Inspection Service, 

Washington, DC, USA. 

European pharmacopoeia. 2012 Evaluation of 
efficacy of veterinary vaccines and immunosera. 
Ch. 5.2.7, 8th edn. Version 8.2. 



17. Council Directive. 2001 2001/82/EC - Directive 2001/ 
82/EC of the European Parliament and of the 
Council of 6 November 2001 on the Community 
code relating to veterinary medicinal products 
(Official Joumal L 311, 28/11/2001 p. 1-66). 

18. Hope T, McMillan J. 2004 Challenge studies of 
human volunteers: ethical issues. / Med Ethics 30, 
110-116. (doi:1 0.1 1 36/jme.2003. 004440) 

19. Henry JL, Jaikaran ES, Davies JR, Tomlinson AJ, 
Mason PJ, Bames JM, Beale AJ. 1966 A study of 
poliovaccination in infancy: excretion following 
challenge with live virus by children given killed or 
living poliovaccine. J. Hyg. (Lond) 64, 105-120. 
(doi:10.1017/S0022172400040389) 

20. Bonell CP, Hargreaves J, Cousens S, Ross D, Hayes R, 
Petticrew M, Kirkwood BR. 2011 Alternatives to 
randomisation in the evaluation of public health 
interventions: design challenges and solutions. 

i. Epidemiol. Community Health 65, 582-587. 
(doi:10.1136/jech.2008.082602) 

21. European Medicines Agency. 2001 Committee for 
veterinary medicinal products: note for guidance 
field trials with veterinary vaccines. EMEA/CVMP/ 
852/99-FINAL. 

22. Clemens J, Brenner R, Rao M, Tafari N, Lowe C. 1996 
Evaluating new vaccines for developing countries. 
Efficacy or effectiveness? 275, 390-397. (doi:10. 
IOOI/jama.1996.03530290060038) 

23. Orenstein WA, Bemier RH, Hinman AR. 1988 
Assessing vaccine efficacy in the field. Further 
observations. Epidemiol. Rev. 10, 

24. Orenstein WA, Bemier RH, Dondero TJ, Hinman AR, 
Marks JS, Bart KJ, Sirotkin B. 1985 Field evaluation 
of vaccine efficacy. Bull. World Health Organ. 63, 
1055-1068. 

25. Hogerwerf L. 2011 Reduction of Coxiella burnetii 
prevalence by vaccination of goats and sheep, the 
Netherlands. Emerg. Infect. Dis. 17, 379-386. 
(doi:10.3201/eid1703.101157) 

26. Knight-Jones TJ, Bulut AN, Gubbins S, Stark KD, 
Pfeiffer DU, Sumption KJ, Paton DJ. 2014 
Retrospective evaluation of foot-and-mouth disease 
vaccine effectiveness in Turkey. Vaccine 32, 
1848-1855. (doi:10.1016/j.vaccine.2014.01.071) 

27. Fine PE, Ponnighaus JM, Maine N, Clarkson JA, Bliss 
L. 1986 Protective efficacy of BCG against leprosy in 
Northem Malawi. Lancet 2, 499-502. (doi:10.1016/ 
S0140-6736(86)90367-3) 

28. Marin M, Nguyen HQ, Langidrik JR, Edwards R, 
Briand K, Papania MJ, Seward JF, LeBaron CW. 2006 
Measles transmission and vaccine effectiveness 
during a large outbreak on a densely populated 
island: implications for vaccination policy. Clin. 
Infect. Dis. 42, 315-319. (doi:1 0.1 086/498902) 

29. Leval A et al. 2013 Quadrivalent human 
papillomavirus vaccine effectiveness: a Swedish 
national cohort study. J. Natl Cancer Inst. 105, 
469 - 474. (doi:1 0.1 093/jnci/djt032) 

30. Grassly NC, Wenger J, Durrani S, Bahl S, Deshpande 
JM, Sutter RW, Heymann DL, Aylward RB. 2007 



Protective efficacy of a monovalent oral type 1 
poliovirus vaccine: a case -control study. Lancet 
369, 1356-1362. (doi:10.1016/s0140-6736(07) 
60531-5) 

31. O'Loughlin RE, Edmond K, Mangtani P, Cohen AL, 
Shetty S, Hajjeh R, Mulholland K. 2010 
Methodology and measurement of the effectiveness 
of Haemophilus influenzae type b vaccine: 
systematic review. Vaccine 28, 6128-6136. (doi:10. 
1016/j.vaccine.2010.06.107) 

32. Feikin DR, Scott JA, Gessner BD. 2014 Use of 
vaccines as probes to define disease burden. Lancet. 
(doi:10.1016/S0140-6736(13)61682-7) 

33. Brown CA, Lilford RJ. 2006 The stepped wedge trial 
design: a systematic review. BMC Med Res. 
Methodol. 6, 54. (doi:10.1 186/1471-2288-6-54) 

34. Swayne DE. 2009 Avian influenza vaccines and 
therapies for poultry. Comp. Immunol Microbiol 
Infect. Dis. 32, 351-363. (doi:10.1016/j.cimid.2008. 
01.006) 

35. Miller E, Salisbury D, Ramsay M. 2001 Planning, 
registration, and implementation of an 
immunisation campaign against meningococcal 
serogroup C disease in the UK: a success story. 
Vaccine 20(Suppl. 1), S58-67. (doi:10.1016/S0264- 
410X(01)00299-7) 

36. Frasch CE, Preziosi MP, LaForce FM. 2012 
Development of a group A meningococcal conjugate 
vaccine, MenAfriVac(TM). Hum. Vaccin. 
Immunotherapeutics 8, 715-724. (doi:10.4161/ 
hv.19619) 

37. Midtlyng PJ et al. 2011 Three Rs approaches in the 
production and quality control of fish vaccines. 
Biologicals 39, 117-128. (doi:10.1016/j.biologicals. 
2011.02.001) 

38. Reeve R, Cox S, Smitsaart E, Beascoechea CP, Haas 
B, Maradei E, Haydon DT, Bamett P. 2011 Reducing 
animal experimentation in foot-and-mouth disease 
vaccine potency tests. Vaccine 29, 5467-5473. 
(doi:10.1016/j.vaccine.201 1.05.056) 

39. Paton DJ, Valarcher JF, Bergmann I, Matlho OG, 
Zakharov VM, Palma EL, Thomson GR. 2005 
Selection of foot and mouth disease vaccine 
strains— a review. Rev. Sci. Tech. 24, 981-993. 

40. Robiolo B, La Torre J, Duffy S, Leon E, Seki C, Torres 
A, Mattion N. 2010 Quantitative single serum- 
dilution liquid phase competitive blocking ELISA for 
the assessment of herd immunity and expected 
protection against foot-and-mouth disease virus in 
vaccinated cattle. J. Virol Methods 166, 21-27. 
(doi:10.1016/j.jviromet.2010.02.011) 

41 . PNEFA. 2007 Assessment of the population immunity 
deriving from the vaccination campaigns against foot 
and mouth disease. Brasil. 

42. WHO. 2012 Recommended composition of influenza 
virus vaccines for use in the 2012-2013 northern 
hemisphere influenza season. Report of strain 
selection meeting February 2012. See http://www. 
who.int/influenza/vaccines/virus/recommendations/ 
201202_recommendation.pdf. 



b 

00 



OO 
VO 



43. Skowronski DM et al. 2013 Interim estimates of 47. 
influenza vaccine effectiveness in 2012/13 from 
Canada's sentinel surveillance network, January 

2013. Euro surveillance: bulletin Europeen sur les 
maladies transmissibles= European communicable 
disease bulletin, 18. 

44. Brehm KE, Kumar N, Thuike HH, Haas B. 2008 High 
potency vaccines induce protection against 
heterologous challenge with foot-and-mouth 48. 
disease virus. Vaccine 26, 1681-1687. (doi:10. 
1016/j.vaccine.2008.01.038) 

45. Thakur A, Pedersen LE, Jungersen G. 2012 Immune 
markers and correlates of protection for vaccine 

induced immune responses. Vaccine 30, 4907- 49. 
4920. (doi:1 0.1 01 6/j.vaccine.201 2.05.049) 

46. Fletcher HA. 2007 Correlates of immune protection 
from tuberculosis. Curr. MoL Med. 7, 319-325. 
(doi:10.2174/156652407780598520) 



Wedlock DN, Denis M, Vordermeier HM, Hewinson 50. 
RG, Buddie BM. 2007 Vaccination of cattle with 
Danish and Pasteur strains of Mycobacterium bovis 
BCG induce different levels of IFN7 post- 
vaccination, but induce similar levels of protection 51. 
against bovine tuberculosis. Vet. Immunol. 
Immunopathol. 118, 50-58. (doi:10.1016/j.vetimm. 
2007.04.005) 

Lesellier S et al. 2011 Protection of Eurasian 
badgers {Meles meles) from tuberculosis after intra- 
muscular vaccination with different doses of BCG. 
Vaccine 29, 3782-3790. (doi:10.1016/j.vaccine. 52. 
2011.03.028) 

Reeve R et al. 2010 Sequence-based prediction for 
vaccine strain selection and identification of 53. 
antigenic variability in foot-and-mouth disease 
virus. PLoS Comput. Biol. 6, e1001027. (doi:10.1371/ 
joumal.pcbi.1001027) 



Shoemaker CA, Klesius PH, Drennan JD, Evans JJ. 
2011 Efficacy of a modified live Flavobacterium 
columnare vaccine in fish. Fish Shellfish Immunol. 
30, 304-308. (doi:10.1016/j.fsi.2010.11.001) 
Cull CA, Paddock ZD, Nagaraja TG, Bello NM, 
Babcock AH, Renter DG. 2012 Efficacy of a 
vaccine and a direct-fed microbial against fecal 
shedding of Escherichia coll 0157:H7 in a 
randomized pen-level field trial of commercial 
feedlot cattle. Vaccine 30, 6210-6215. (doi:10. 
1 01 6/j.vaccine.201 2.05.080) 
WHO. 2005 Immunization coverage cluster 
survey — reference manual. See http://www.who. 
int/vaccines-documents/DocsPDF05/www767.pdf. 
Fine PE. 1995 Variation in protection by BCG: 
implications of and for heterologous immunity. 
Lancet 346, 1339-1345. (doi:10.1016/S0140- 
6736(95)92348-9) 



o 

00 



00 



